Method of controlling device and electronic device

ABSTRACT

An input of a command for controlling an electronic device is provided. The method of operating the electronic device includes determining at least one anchor point by designating a target to be controlled, based on a first input, identifying an operation to be performed for the target designated by the at least one anchor point, based on a second input, and performing the operation.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. §119(a) of a Korean patent application number 10-2019-0008756, filed onJan. 23, 2019, in the Korean Intellectual Property Office, thedisclosure of which is incorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to an electronic device. More particularly, thedisclosure relates to a method of controlling an electronic device andthe electronic device.

2. Description of Related Art

In general, a portable electronic device, such as a smart phone includesa touch screen as an input means. The touch screen is able tointuitively interact with a user in that the user can make an input on ascreen. However, the size of an area for making a touch input may belimited according to the size of the electronic device in which case theaccuracy of the touch input using a finger may have limits. Further, inthe case of services based on a technology, such as Virtual Reality (VR)or Augmented Reality (AR), there is no means for the touch input or thetouch input may be made beyond a user's sight.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providea method and an electronic device for inputting a command for moreeffectively controlling a device.

Another aspect of the disclosure is to provide a method and anelectronic device for controlling a device using other means as well asa touch input.

Another aspect of the disclosure is to provide a method and anelectronic device for controlling a device through a voice, an eye gaze,or a gesture in a wireless communication system.

Another aspect of the disclosure is to provide a method and anelectronic device for controlling an object displayed on a devicethrough a voice, an eye gaze, or a gesture in a wireless communicationsystem.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, a method performed by anelectronic device is provided. The method includes determining at leastone anchor point by designating a target to be controlled, based on afirst input, identifying an operation to be performed for the targetdesignated by the at least one anchor point, based on a second input,and performing the operation.

In accordance with another aspect of the disclosure, an electronicdevice is provided. The electronic device includes a detector configuredto detect a state inside or outside the electronic device and at leastone processor connected to the detector. The at least one processor maydetermine at least one anchor point by designating a target to becontrolled based on a first input identified by the detector, identifyan operation to be performed for the target designated by the at leastone anchor point based on a second input identified by the detector, andperform the operation.

A method and an electronic device according to various embodiments ofthe disclosure can input a command for controlling a device using avoice, an eye gaze, and a gesture, thereby more easily controlling thedevice.

Effects which can be acquired by the disclosure are not limited to theabove described effects, and other effects that have not been mentionedmay be clearly understood by those skilled in the art from the followingdescription.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a block diagram of an electronic device according to anembodiment of the disclosure;

FIG. 2 is a flowchart illustrating an operation of an electronic deviceaccording to an embodiment of the disclosure;

FIG. 3 illustrates a head movement which can be detected by anelectronic device according to an embodiment of the disclosure;

FIG. 4A is a flowchart illustrating a process in which an electronicdevice modifies a parameter related to an object based on various typesof inputs according to an embodiment of the disclosure;

FIG. 4B illustrates a change in a head movement and an object-relatedparameter according to a lapse of time in an electronic device accordingto an embodiment of the disclosure;

FIG. 5 is a flowchart illustrating a process in which an electronicdevice controls an object within a virtual space based on an eye gazeand head rotation according to an embodiment of the disclosure;

FIG. 6A is a flowchart illustrating a process in which an electronicdevice modifies an image based on an eye gaze and a touch inputaccording to an embodiment of the disclosure;

FIG. 6B illustrate images modified using an eye gaze and a touch inputin an electronic device according to an embodiment of the disclosure;

FIG. 7A is a flowchart illustrating a process in which an electronicdevice rotates an image based on an eye gaze and a touch input accordingto an embodiment of the disclosure;

FIG. 7B illustrates an image rotated using an eye gaze and a touch inputin an electronic device according to an embodiment of the disclosure;

FIG. 8 is a flowchart illustrating a process in which an electronicdevice controls a zoom level of a camera based on an eye gaze and headrotation according to an embodiment of the disclosure;

FIG. 9 is a flowchart illustrating a process in which an electronicdevice controls a focus of a camera based on an eye gaze according to anembodiment of the disclosure;

FIG. 10 is a flowchart illustrating a process in which an electronicdevice controls an object without eye gaze tracking according to anembodiment of the disclosure;

FIG. 11A is a flowchart illustrating a process in which an electronicdevice controls an Augmented Reality (AR) object using an eye gazeaccording to an embodiment of the disclosure;

FIG. 11B illustrates an eye gaze for an object in a real space and an ARobject in an electronic device according to an embodiment of thedisclosure;

FIG. 12A illustrates an input which can be recognized by a VirtualReality (VR) headset in an electronic device according to an embodimentof the disclosure;

FIG. 12B illustrates an input which can be recognized by a VirtualReality (VR) headset in an electronic device according to an embodimentof the disclosure;

FIG. 13A is a flowchart illustrating a process in which an electronicdevice moves a viewpoint within a VR space according to an embodiment ofthe disclosure;

FIG. 13B illustrates a viewpoint movement within a VR space in anelectronic device according to an embodiment of the disclosure;

FIG. 13C illustrates a viewpoint movement within a VR space in anelectronic device according to an embodiment of the disclosure

FIG. 13D illustrates a viewpoint movement within a VR space in anelectronic device according to an embodiment of the disclosure;

FIG. 14A is a flowchart illustrating a process in which an electronicdevice controls an object within a VR space according to an embodimentof the disclosure;

FIG. 14B illustrates an object control within a VR space in anelectronic device according to an embodiment of the disclosure;

FIG. 14C illustrates an object control within a VR space in anelectronic device according to an embodiment of the disclosure;

FIG. 14D illustrates an object control within a VR space in anelectronic device according to an embodiment of the disclosure;

FIG. 15A is flowchart illustrating a process in which an electronicdevice rotates an AR object according to an embodiment of thedisclosure;

FIG. 15B illustrates a rotation of an AR object in an electronic deviceaccording to an embodiment of the disclosure;

FIG. 16A is a flowchart illustrating a process in which an electronicdevice rotates an AR object according to an embodiment of thedisclosure; and

FIG. 16B illustrates a rotation of an AR object in an electronic deviceaccording to an embodiment of the disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

The terms used in the disclosure are only used to describe specificembodiments, and are not intended to limit the disclosure. A singularexpression may include a plural expression unless they are definitelydifferent in a context. Unless defined otherwise, all terms used herein,including technical and scientific terms, have the same meaning as thosecommonly understood by a person skilled in the art to which thedisclosure pertains. Such terms as those defined in a generally useddictionary may be interpreted to have the meanings equal to thecontextual meanings in the relevant field of art, and are not to beinterpreted to have ideal or excessively formal meanings unless clearlydefined in the disclosure. In some cases, even the term defined in thedisclosure should not be interpreted to exclude embodiments of thedisclosure.

Hereinafter, various embodiments of the disclosure will be describedbased on an approach of hardware. However, various embodiments of thedisclosure include a technology that uses both hardware and software andthus, the various embodiments of the disclosure may not exclude theperspective of software.

Hereinafter, the disclosure provides a method of controlling a deviceand an electronic device therefor. Specifically, the disclosuredescribes a technology for inputting a command for controlling a deviceby making various types of inputs, such as a voice, an eye gaze, and agesture into an electronic device.

The terms referring to a control target used by an electronic device inthe following description, the terms referring to a state of the controltarget, the terms referring to a user's gesture, and the terms referringto an element of a device are mentioned for convenience of description.Accordingly, the disclosure is not limited to the following terms andother terms having the same technical meaning may be used.

Further, the expression “larger than” or “smaller than” is used todetermine whether a specific condition is satisfied or fulfilled, but isonly to indicate an example and does not exclude “larger than or equalto” or “equal to or smaller than” A condition indication “larger than orequal to” may be replaced with a condition indicating “larger than”, acondition indicating “equal to or smaller than” may be replaced with“smaller than”, and a condition indicating “larger than or equal to andsmaller than” may be replaced with “larger than and smaller than”.

FIG. 1 is a block diagram of an electronic device according to anembodiment of the disclosure.

Referring to FIG. 1 , an electronic device 100 includes a display unit110, an input unit 120, a detector 130, a storage unit 140, and acontroller 150.

The display unit 110 visually outputs graphics and/or characters. Thedisplay unit 110 may visually provide information to the outside (forexample, a user) according to the control of the controller 150. Forexample, the display unit 110 may include a display, a hologram device,or a projector, and a control circuit for controlling the correspondingdevice. For example, the display unit 110 may include at least one of aLiquid Crystal Display (LCD), a Light Emitting Diode (LED), a LightEmitting Polymer Display (LPD), an Organic Light Emitting Diode (OLED),an Active Matrix Organic Light Emitting Diode (AMOLED), or a FlexibleLED (FLED). Further, in order to provide a Virtual Reality (VR) orAugmented Reality (AR) service, the electronic device 100 may include anoptical member (for example, a lens) when the electronic device 100 is ahead-mounted type or a glass type.

The input unit 120 detects an input from the outside (for example, theuser). For example, the input unit 120 may identify a command or data tobe used by an element of the electronic device 100 from the outside (forexample, the user) of the electronic device 100. The input unit 120 mayinclude, for example, a microphone, a mouse, a keyboard, a keypad, abutton, a touch pad, or a digital pen (for example, a stylus pen).According to an embodiment, the input unit 120 and the display unit 110may be configured as a single module. In this case, the input unit 120may be a touch circuitry or a touch panel for detecting a touch, and thedisplay unit 110 and the input unit 120 may configure a touch screen.

The detector 130 detects a state of the inside or the outside of theelectronic device 100. In other words, the detector 130 generates asignal or data corresponding to the state of the inside or the outsideof the electronic device 100. The detector 130 may include hardware forgenerating an electric signal in response to a physical change (forexample, light, magnetic field, gravity, and temperature). For example,the detector 130 may include at least one of a camera, a gravity sensor,a gyro sensor, an acceleration sensor, a gesture sensor, a proximitysensor, a color sensor, an infrared sensor, a biometric sensor, ahumidity sensor, and an illumination sensor.

The storage unit 140 may store various pieces of data used by at leastone element of the electronic device 100. The data may include, forexample, software and input data or output data related thereto. Thestorage unit 140 may include volatile memory or nonvolatile memory.

The controller 150 controls the overall operation of the electronicdevice 100. For example, the controller 150 may control at least oneother element (for example, a hardware or software element) of theelectronic device 100 connected to the controller 150 by executingsoftware and perform various data processing or calculations. Accordingto an embodiment, as at least a portion of the data processing orcalculations, the controller 150 may load a command or data from anotherelement to the storage unit 140, process the command or data stored inthe storage unit 140, and store resultant data in the storage unit 140.To this end, the controller 150 may include a main processor (forexample, a central processing unit or an application processor) and anauxiliary processor (for example, a graphic processing unit, an imagesignal processor, a sensor hub processor, or a communication processor)that can operate independently or together therewith.

According to various embodiments, the controller 150 may recognize auser's command input through various types and control the electronicdevice 100. For example, the user's command may be input through varioustypes, such as a touch input, an eye gaze, a voice, and a gesture. Theuser's command may include an object to apply the command and thecontent of the command and may be specified by two or more schemes.According to various embodiments, the controller 150 may control theelectronic device 100 to perform operations according to the followingvarious embodiments.

The electronic device may interact with the user. For example, the usermay provide a command through an input means (for example, the inputunit 120) included in the electronic device, and the electronic devicemay perform a function corresponding to the command. In general, a touchscreen is widely used as the input means.

However, it is not easy to control a complex application through thetouch input alone. For example, when an image editing application isused in a mobile environment, very accurate and delicate control is noteasy due to the size of the screen and the accuracy of a finger gesture.In most cases, an application function does not depend on the screensize, and available characteristics are the same in 3-inch, 5-inch, andtablet devices. In many cases, a television (TV) has the samecharacteristic and is controlled in the same way as small mobiledevices.

The electronic device may be used for the VR or AR service in ahead-mounted type or a glass type in which case a touch cannot begenerally used. For example, in the case of VR or AR, a display unit(for example, the display unit 110) of the electronic device is veryclose to eyes, so that the user may not use a touch screen of theelectronic device to control the screen (scene).

As described above, when controlling the electronic device, there arelimits on the control due to the screen size, the size of fingertips, alack of the touch screen, or an impossibility of the use of the touchscreen. Accordingly, the disclosure proposes various embodiments of moreeffectively controlling the electronic device.

FIG. 2 is a flowchart illustrating an operation of an electronic deviceaccording to an embodiment of the disclosure. FIG. 2 illustrates amethod of operating the electronic device 100.

Referring to FIG. 2 , in operation 201, the electronic device starts acontrol mode. The control mode is a mode in which the electronic devicecan be controlled based on inputs in two or more forms. According tovarious embodiments, the control mode may be started, entered, oractivated by a predefined start command. For example, the start commandmay be defined based on an input in at least one form. Specifically, thestart command may be defined as a voice having the predetermined contentor a button input in a predefined pattern. The start command may bereferred to as a “start action” or another term having a technicalmeaning equivalent thereto.

In operation 203, the electronic device determines an anchor point. Theanchor point is an object in a virtual world or a real world displayedon the electronic device or one point on an object existing in a realworld and is used to designate a target, to which a function to beperformed later is applied, that is, a target to be controlled in thecontrol mode. For example, the target to be controlled may be one of anobject, a specific point of the object, a space, and a specific locationof the space. According to various embodiments, the anchor point may bedetermined based on an input in at least one form. For example, theinput in at least one form may include at least one of voice detection(for example, a linguistic meaning of a voice, a voice tone, and a voicepitch), eye gaze tracking (for example, a location which an eye gazefaces or a time during which an eye gaze is maintained), a touch input(for example, a touch position, a touch type, a touch time, or a touchpattern), and a button input (for example, a button type, a press time,or a press pattern). According to an embodiment, an input fordetermining the anchor point may be defined or detected jointly with thestart command.

In operation 205, the electronic device identifies the control command.The control command indicates an operation to be performed for a targetdesignated by the anchor point. According to various embodiments, thecontrol command may be identified based on an input in at least oneform. For example, the input in at least one form may include at leastone of a gesture (for example, hand movement, eye blink, facialexpression, head rotation, body movement, or a change in a position ofthe electronic device), voice detection (for example, a linguisticmeaning of a voice, a voice tone, or a voice pitch), gaze tracking (forexample, a location which an eye gaze faces or a time during which aneye gaze is maintained), a touch input (for example, the number of touchinputs or a pattern of a touch input), and a button input. According toan embodiment, the input in one form may be used to identify a functiontype (for example, rotation, movement, stretching/pulling, or change),and the input in the other form may be used to identify a parameter ofthe function (for example, a direction of rotation, a direction ofmovement, a direction and a degree of stretching/pulling, and a degreeof a change).

In operation 207, the electronic device performs a functioncorresponding to the control command. The content of the control commanddepends on an object designated by the anchor point, a control commanddetected after the anchor point is determined, and a currently executedapplication. The function corresponding to the control command may beperformed based on the anchor point or performed for the objectdesignated by the anchor point.

In operation 209, the electronic device ends the control mode. Accordingto an embodiment, the control mode may be ended or released by apredefined end command. For example, the end command may be definedbased on an input in at least one form. Specifically, the end commandmay be defined as a voice having the predefined content or atouch/button input in a predefined pattern. The end command may bereferred to as an “end action” or another term having a technicalmeaning equivalent thereto. According to another embodiment, the controlmode may be ended by the lapse of a predetermined time from the entryinto the control mode or the lapse of a predetermined time aftergeneration of the control command.

Referring to FIG. 2 , the electronic device may perform a correspondingfunction through various types of inputs during the control mode. Inorder to recognize various types of inputs, a capability for detecting atouch, a voice, a gesture, or an eye gaze may be needed. Specifically,various types of inputs may include at least one of a finger (forexample, an operation for an image, such as the size control, adisplayed three-dimensional (3D) object or an avatar, or an operationfor a business application within a VR scene or an Excel file), eye gazeand additional input information (for example, the eye gaze can betracked by a built-in camera), and head tracking (for example, movementor rotation). Rotation of the head which can be detected by headtracking may be as illustrated in FIG. 3 below.

FIG. 3 illustrates a head movement which can be detected by anelectronic device according to an embodiment of the disclosure.

Referring to FIG. 3 , rotation of the head with respect to an axisfacing upwardly is defined as yaw 310, rotation with respect to an axistoward the face is defined as roll 320, and rotation with respect to anaxis facing the lateral side of the face is defined as pitch 330. Therotation of the head may be tracked based on an image captured by thebuilt-in camera or based on a measurement value of another sensor (forexample, a gyro sensor or an Inertial Navigation System (INS)).

FIG. 4A is a flowchart illustrating a process in which an electronicdevice modifies a parameter related to an object based on various typesof inputs according to an embodiment of the disclosure. FIG. 4Aillustrates a method of operating the electronic device 100.

FIG. 4B illustrates a change in a head movement and an object-relatedparameter according to a lapse of time in an electronic device accordingto an embodiment of the disclosure.

Referring to FIG. 4A, in operation 401, the electronic device identifieswhether a start command is detected. According to an embodiment, theuser may use a voice command in order to turn on a control mode. Forexample, referring to FIG. 4B, a start command 402 may be detected at atime point t₂.

The electronic device tracks a finger gesture in operation 403. Theelectronic device tracks an eye gaze in operation 405. The electronicdevice tracks head rotation in operation 407. The electronic device maydetermine an anchor point through operations 403, 405, and 407.According to an embodiment, the anchor point may be identified bytracking one point of an object or an eye gaze at an object to becontrolled. The anchor point may be determined once during the overallprocedure or continuously determined. According to an embodiment, theelectronic device may determine a control command based on the fingergesture or the head rotation. Although not illustrated in FIG. 4A,another type of the input (for example, a voice, a button, or a facialexpression) may be further used to determine the control command. Forexample, referring to FIG. 4B, a control command 404 may be identifiedby head movement starting at a time point t₂.

In operation 409, the electronic device modifies a parameter related tothe object. According to an embodiment, the electronic device may modifythe parameter related to the object based on a finger or head location.For example, referring to FIG. 4B, the parameter related to the objectmay be modified from a time point t₃ based on a control command 404generated at the time point t₁ and head movement detected from the timepoint t₂. Thereafter, when an amount of the change in head movement ischanged at a time point t₄ and a time point t₅, an amount of the changein the parameter may also be changed.

In operation 411, the electronic device detects an end command.Accordingly, the control mode is terminated. The user may use a voicecommand for turning off the control mode. For example, referring to FIG.4B, an end command 406 may be generated between the time point t₅ and atime point t₆. Accordingly, after the time point t₆, the head movementmay be analyzed and processed according to a rule defined in a modeother than the control mode.

The operations described with reference to FIG. 4A may be sequentiallyperformed. According to another embodiment, some operations may beperformed in parallel. For example, measurement of eye gaze trackingand/or tracking of the head location may be performed before the controlmode starts. According to another embodiment, at least one operation maybe omitted or replaced. For example, the voice command may be replacedwith another input for turning on/off.

As described above, the electronic device according to variousembodiments may designate an object based on various types of inputs andcontrol the designated object. The object may be one of content,attributes of the content, an application, a parameter related to theapplication, and an element of the electronic device. Variousembodiments may be derived by selecting various types of inputsdescribed in the above embodiment for each operation. Hereinafter, thedisclosure describes more detailed embodiments. In the followingembodiments, the description of the start command and the end command isomitted but it is apparent that the operation for starting and endingthe control mode by the start command and the end command may be furtherperformed.

FIG. 5 is a flowchart illustrating a process in which an electronicdevice controls an object within a virtual space based on an eye gazeand head rotation according to an embodiment of the disclosure. FIG. 5illustrates a method of operating the electronic device 100.

Referring to FIG. 5 , in operation 501, the electronic device displays aVR screen. The electronic device may display a VR screen in the state inwhich the electronic device is equipped with an auxiliary device (forexample, headset). The VR screen may include at least one object (forexample, virtual model).

In operation 503, the electronic device identifies whether an eye gazeat an object is maintained for a predetermined time or longer. Theelectronic device tracks an eye gaze of the user through a camera andidentifies whether the eye gaze at a specific object is maintained for apredetermined time or longer, in other words, the time for which the eyegaze at the specific object is maintained is larger than a thresholdvalue according to the tracking result. Whether the eye gaze ismaintained may be determined based on the entire surface of the objector based on a partial area on the object. When the eye gaze at theobject is maintained for a predetermined time or longer, one point onthe surface of the corresponding object or inside the correspondingobject or one nearby point may be determined as an anchor point.

When the eye gaze at the object is maintained for a predetermined timeor longer, the electronic device identifies whether head rotation isdetected in operation 505. The head rotation may be identified by threecomponents, such as yaw, pitch, and roll. The electronic device maydetect head rotation through a camera or a sensor.

When the head rotation is detected, the electronic device rotates theobject within the virtual space according to a rotation direction of thehead in operation 507. The electronic device rotates the object based onthe anchor point. At this time, the electronic device may rotate theobject in a direction, which is the same as the direction of headrotation or a direction obtained by converting the direction of headrotation according to a predefined rule. According to anotherembodiment, the anchor point may be continuously changed. For example,the anchor point may be periodically determined based on a point whichthe eye gaze faces.

In the embodiment described referring to FIG. 5 , the electronic devicedisplays the VR screen and then identifies whether the eye gaze at theobject is maintained for a predetermined time or longer. According toanother embodiment, when a start command is detected after the VR screenis displayed, the electronic device may perform operation 503. In thiscase, if there is no start command, the electronic device may analyzethe head rotation as a navigation command. According to an embodiment,the electronic device may deactivate a navigation function according tothe start command and analyze the head rotation as an object rotationcommand. According to another embodiment, the electronic device mayanalyze only some of the three components (for example, yaw, pitch, androll) included in the head rotation as the object rotation command andanalyze the remaining components are the navigation command. A methodrelated to the start command may be applied to another embodiment.

When the start command is used, intervals in which the maintenance ofthe eye gaze can be recognized may be variously defined. According to anembodiment, the maintenance of the eye gaze within an interval beforethe start command may be excluded from the comparison with a thresholdvalue. According to another embodiment, the maintenance of the eye gazewithin the interval before the start command may be considered whencompared with the threshold value. In this case, the time for which theeye gaze is maintained may include a time before and after the startcommand. For example, the electronic device may identify a start pointof the time for which the eye gaze is maintained within an intervalafter the start command or identify the same within an intervalincluding a time before the start command. A method related to the startcommand may be applied to another embodiment.

FIG. 6A is a flowchart illustrating a process in which an electronicdevice modifies an image based on an eye gaze and a touch inputaccording to an embodiment of the disclosure. FIG. 6A illustrates amethod of operating the electronic device 100.

FIG. 6B illustrate images modified using an eye gaze and a touch inputin an electronic device according to an embodiment of the disclosure.

Referring to FIG. 6A, in operation 601, the electronic device displaysat least one image. For example, the electronic device may execute animage editing application and display at least one image.

In operation 603, the electronic device identifies whether an eye gazeat a specific point of the image is maintained for a predetermined timeor longer. The electronic device tracks the eye gaze of the user throughthe camera and identifies whether the eye gaze at a specific point ismaintained for a predetermined time or longer, in other words, the timefor which the eye gaze at the specific point is larger than a thresholdvalue according to the tracking result. When the eye gaze at thespecific point is maintained for a predetermined time or longer, thecorresponding point may be determined as an anchor point.

When the eye gaze at the specific point is maintained for apredetermined time or longer, the electronic device identifies whether atouch input for modifying the image is detected in operation 605. Forexample, image modification may be rotation, size control,stretching/pulling, distortion, filtering (or blurring) or anotherediting function.

When the touch input for the image modification is detected, theelectronic device may modify the image based on the anchor point inoperation 607. For example, when a blurring effect is provided, theelectronic device may form a boundary at a point at which a touch input622 a of the user is detected and blur an area outside the boundarybased on the anchor point 620 as illustrated in FIG. 6B. Alternatively,the electronic device may control a slide bar based on the touch input622 b of the user, determine a distance between the boundary and theanchor point 620 according to a state of the slide bar, and then blurthe area outside the boundary based on the anchor point 620. In anotherexample, when image stretching/pulling is performed, the electronicdevice may move corners of the original image to points at which touchinputs 632 a, 632 b, 632 c, and 632 d are generated based on the anchorpoint 620 as illustrated in FIG. 6C.

In the embodiment described referring to FIG. 6A, the anchor point isdetermined by maintenance of an eye gaze at a specific point for apredetermined time or longer. According to another embodiment, theanchor point may be determined by a voice command without anydetermination about whether the eye gaze is maintained for apredetermined time. For example, when the user gazes at a specific pointand makes a predefined voice command (for example, “Fix!”), theelectronic device may determine a point at which the user gazes when thevoice command is generated as the anchor point. A method of determiningthe anchor point based on the eye gaze and the voice command may beapplied to another embodiment.

FIG. 7A is a flowchart illustrating a process in which an electronicdevice rotates an image based on an eye gaze and a touch input accordingto an embodiment of the disclosure. FIG. 7A illustrates a method ofoperating the electronic device 100.

FIG. 7B illustrates an image rotated using an eye gaze and a touch inputin an electronic device according to an embodiment of the disclosure.

Referring to FIG. 7A, in operation 701, the electronic device displaysat least one image. For example, the electronic device may execute anapplication related to the image (for example, an image displayapplication or an image editing application) and display at least oneimage.

In operation 703, the electronic device identifies whether an eye gazeat a specific point of the image is maintained for a predetermined timeor longer. The electronic device tracks the eye gaze of the user throughthe camera and identifies whether the eye gaze at a specific point ismaintained for a predetermined time or longer, in other words, the timefor which the eye gaze at the specific point is larger than a thresholdvalue according to the tracking result. When the eye gaze at thespecific point is maintained for a predetermined time or longer, thecorresponding point may be determined as an anchor point.

When the eye gaze at the specific point is maintained for apredetermined time or longer, the electronic device identifies whetherhead rotation is detected in operation 705. The head rotation may beidentified by three components, such as yaw, pitch, and roll. Theelectronic device may detect head rotation through a camera or a sensor.

When head rotation is detected, the electronic device rotates the imageaccording to a head rotation direction in operation 707. For example,the electronic device analyzes the head rotation as an image rotationcommand. For example, when the eye gaze of the user is maintained at apoint 720 for a predetermined time or longer and then the user rotatesthe head as illustrated in FIG. 7B, the image may be rotated accordingto a head rotation direction. At this time, a reference point of therotation may be a predefined point or a point (for example, the point720) determined by the eye gaze of the user. When the reference point isdetermined by the eye gaze of the user, the reference point may bedetermined when the image is selected (for example, in operation ofoperation 703) or determined by an eye gaze direction after the image isspecified.

In the embodiment described referring to FIG. 7A, the image is rotatedaccording to head rotation. According to another embodiment, the headrotation may be replaced with a voice command. For example, when theuser makes a voice command (for example, “Rotate left!”, “Rotate up!”)for rotating the image, the electronic device may detect a directionincluded in the voice command and rotate the image.

FIG. 8 is a flowchart illustrating a process in which an electronicdevice controls a zoom level of a camera based on an eye gaze and headrotation according to an embodiment of the disclosure. FIG. 8illustrates a method of operating the electronic device 100.

Referring to FIG. 8 , in operation 801, the electronic device executes acamera application. The electronic device activates a camera moduleaccording to execution of the camera application and displays a previewimage of an input made through the camera module.

In operation 803, the electronic device identifies whether an eye gazeis maintained at a specific point of the preview image for apredetermined time or longer. The electronic device tracks the eye gazeof the user through the camera and identifies whether the eye gaze at aspecific point is maintained for a predetermined time or longer, inother words, the time for which the eye gaze at the specific point islarger than a threshold value according to the tracking result. When theeye gaze at the specific point is maintained for a predetermined time orlonger, the corresponding point may be determined as an anchor point.

When the eye gaze at the specific point is maintained for apredetermined time or longer, the electronic device identifies whetherhead rotation is detected in operation 805. The head rotation may beidentified by three components, such as yaw, pitch, and roll. Theelectronic device may detect head rotation through a camera or a sensor.

When head rotation is detected, the electronic device controls the zoomlevel of the camera, that is, zooms in or zooms out according to a headrotation direction in operation 807. For example, the electronic deviceanalyzes the head rotation as a zoom in/output command. When zoom-in/outis performed by digital-zoom, the electronic device may performzoom-in/out based on the anchor point.

In the embodiment described referring to FIG. 8 , when the eye gaze ismaintained for a predetermined time or more, the head rotation istreated as a zoom level control command. According to anotherembodiment, operations 805 and 807 may be performed only through a voicecommand without maintenance of the eye gaze. For example, operation 803may be replaced with an operation for detecting a voice command (forexample, “Control zoom!”) indicating the control of the zoom level byhead rotation.

The embodiment described referring to FIG. 8 is related to zoom-in/outof the camera when the camera application is executed. The embodiment ofFIG. 8 may be expanded to an image gallery. For example, in a situationin which a stored image is displayed rather than an image input throughthe camera, the electronic device may zoom in/out the image through aprocedure similar to operations 803, 805, and 807.

FIG. 9 is a flowchart illustrating a process in which an electronicdevice controls a focus of a camera based on an eye gaze according to anembodiment of the disclosure. FIG. 9 illustrates a method of operatingthe electronic device 100.

Referring to FIG. 9 , in operation 901, the electronic device executes acamera application. The electronic device may execute a camera moduleaccording to execution of the camera application and displays a previewimage of an image input through the camera module.

In operation 903, the electronic device identifies whether an eye gazeis maintained at a specific point of the preview image for apredetermined time or longer. The electronic device tracks the eye gazeof the user through the camera and identifies whether the eye gaze at aspecific point is maintained for a predetermined time or longer, inother words, the time for which the eye gaze at the specific point islarger than a threshold value according to the tracking result.

When an eye gaze is maintained at a specific point for a predeterminedtime or longer, the electronic device controls the focus based on thespecific point in operation 905. For example, the electronic device mayestimate a distance between the electronic device and an object locatedat an anchor point and control the focal distance according to theestimated distance. The eye gaze at the specific point may be analyzedas an input for determining the anchor point, and maintenance of the eyegaze at the specific point for a predetermined time or longer may beanalyzed as a control command for controlling the focus.

In the embodiment described referring to FIG. 9 , the focal distance ofthe camera may be determined by the eye gaze. According to anotherembodiment, the focal distance may be controlled by head rotation. Whenthe user rotates the head after the focal distance is controlled basedon the distance from the object located at the anchor point determinedby the eye gaze, the electronic device may further control the focaldistance according to head rotation. Accordingly, the user may moreaccurately control the focus.

In the above-described various embodiments, the eye gaze is used todetermine the anchor point or the object to be controlled. However,according to other embodiments, the electronic device may determine theanchor point or the object without eye gaze tracking. In this case, eventhough the electronic device does not have an eye gaze trackingfunction, the proposed various embodiments may be implemented. Anembodiment of controlling the object without eye gaze tracking isdescribed with reference to FIG. 10 .

FIG. 10 is a flowchart illustrating a process in which an electronicdevice controls an object without eye gaze tracking according to anembodiment of the disclosure. FIG. 10 illustrates a method of operatingthe electronic device 100.

Referring to FIG. 10 , in operation 1001, the electronic device displaysa screen. The screen may vary depending on an application beingexecuted. For example, the electronic device may display an image, amoving image, and a VR screen. At this time, the electronic devicedisplays a cursor within the screen. According to an embodiment, thecursor may be generated by a start command. In operation 1003, theelectronic device moves the cursor according to movement of the head.The electronic device may detect user head rotation and move the cursoraccording to a direction of the head rotation.

In operation 1005, the electronic device identifies whether a commandfor fixing the cursor is detected. For example, the command for fixingthe cursor may include at least one of a touch, a button press, agesture, and a voice command. When the command for fixing the cursor isdetected, the electronic device fixes the cursor in operation 1007. Forexample, the electronic device fixes the cursor at a location at whichthe cursor is displayed at a time point at which the command for fixingthe cursor is detected. Thereafter, the location at which the cursor isfixed may be treated as an anchor point.

In operation 1009, the electronic device identifies whether headrotation is detected. When the head rotation is detected, the electronicdevice performs a function according to the head rotation in operation1011. The executed function may depend on the executed application orthe location of the anchor point. For example, the electronic device mayperform a function (for example, object rotation, image modification,image rotation, zoom level control, and focus control) related to one ofthe above-described various embodiments.

In the above-described various embodiments, the object is controlledafter the anchor point is determined. According to other embodiments,there may be no separate operation for determining the anchor point.According to an embodiment, the operation for determining the anchorpoint may be combined with the operation for controlling the objectwithout separation therefrom. For example, the anchor point may becontinuously changed while the object is controlled. For example, thecurrent location of the movable cursor may be treated as the anchorpoint while the object is controlled. According to another embodiment,the anchor point may be not variable but may be fixed at a predefinedlocation.

FIG. 11A is a flowchart illustrating a process in which an electronicdevice controls an AR object based on an eye gaze according to anembodiment of the disclosure. FIG. 11A illustrates a method of operatingthe electronic device 100.

FIG. 11B illustrates an eye gaze for an object in a real space and an ARobject in an electronic device according to an embodiment of thedisclosure.

Referring to FIG. 11A, in operation 1101, the electronic deviceidentifies an object in a real space at which the eye gaze of the useris maintained or an object in a real world. The object in the real spaceis displayed on the screen of the electronic device. For example, in thestate in which an application for generating an AR object (for example,an avatar) is executed and a real space is captured through a camera anddisplayed, the electronic device may identify an object of the realspace at which the user gazes based on eye gaze tracking. For example,referring to FIG. 11B, the eye gaze of the user faces an object 1100 aof the real space, and the electronic device 100 may determine that theeye gaze of the user faces the object 1110 a in the real space througheye gaze tracking. At this time, the object 1100 a in the real space maybe captured through the camera, and a captured object 1110 b may bedisplayed on the screen of the electronic device 100.

In operation 1103, the electronic device generates an AR objectcorresponding to the identified object. Since the identified object isdisplayed in the electronic device, the electronic device may detect acharacteristic of the identified object based on the image. Accordingly,the electronic device may generate an AR object including the detectedcharacteristic. For example, the characteristic may include variousattributes, such as an object type (for example, person, dog, cat, andflower) and object appearance (for example, color and size).

In operation 1105, the electronic device identifies whether a commanddesignating a part of the AR object is detected. The command designatingthe part may be defined in various forms. For example, the commanddesignating the part may be identified based on an eye gaze. Accordingto an embodiment, the electronic device may analyze maintenance of theeye gaze at the specific part of the AR object for a predetermined timeor longer as the command designating the part. According to anotherembodiment, the electronic device may analyze a voice command generatedin the state in which the eye gaze is maintained at the specific part ofthe AR object as the command designating the part. For example, when theAR object is a person-shaped avatar, the part which can be designatedmay be hair, face, clothes, hand, or leg.

In operation 1107, the electronic device identifies whether a commandfor changing attributes of the designated part is detected. The commandfor changing the attributes may be defined in various forms. Accordingto an embodiment, the command for changing the attributes may be definedbased on head rotation. According to another embodiment, the command forchanging the attributes may be defined based on movement of a slide barby a touch input.

When the command for changing the attributes is detected, the electronicdevice controls attributes of the AR object according to the detectedcommand in operation 1109. An attribute value indicated by the commandfor changing the attributes may be applied to the AR object, and the ARobject to which the indicated attribute value is applied may bedisplayed. According to an embodiment, when the command for changing theattributes is defined based on the head rotation, if the head is rotatedin one direction in the state in which the part is designated, theelectronic device may apply and display candidate attributes. Forexample, as an angle of rotation increases, a plurality of candidateattributes may be sequentially displayed. According to anotherembodiment, when the command for changing the attributes is definedbased on movement of the slide bar by the touch input, if the slide barmoves according to the touch input in the state in which the part isdesignated, the electronic device may apply and display candidateattributes. For example, as a movement distance of the slide barincreases, the plurality of candidate attributes may be sequentiallydisplayed.

The above-described various embodiments may be expanded to control thetime point or the object in the VR system. For example, theabove-described embodiments may be applied to navigate the user withinthe virtual space, change a camera view of a camera around the object,or translate or rotate the object within the virtual space.

The VR service may be provided by a Head-Mounted Display (HMD)electronic device or an electronic device installed in a headset. Inthis case, since a display is close to a user's eyes, it is generallydifficult to use a touch input through a touch screen. Examples ofinputs which can be recognized through the HMD or the headset aredescribed with reference to FIGS. 12A and 12B.

FIG. 12A illustrates an input which can be recognized by a VR headset ofan electronic device according to an embodiment of the disclosure, andFIG. 12B illustrates an input which can be recognized by a VR headset ofan electronic device according to an embodiment of the disclosure.

Referring to FIG. 12A, an X axis 1211 and a Y axis 1212 may be definedon a touch pad, and a touch input of which a touch point moves along theX axis 1211 and a touch input of which a touch point moves along the Yaxis 1212 may be recognized.

Referring to FIG. 12B, an input 1221 corresponding to up and downrotation of the user's head and an input 1222 corresponding to left andright rotation of the head may be recognized. The control described inthe following embodiment may be performed using the inputs illustratedin FIGS. 12A and 12B.

According to various embodiments, the control operation in the VR spacemay be grouped into lower modes. Each lower mode has a unique purpose.For example, lower modes, such as movement of the user and selection ofrotation of the object in the virtual world may be defined. In eachmode, head rotation corresponds to rotation of a viewpoint in the VRspace

As a first lower mode for the control in the VR mode, user movement,that is, viewpoint movement in the VR space may be processed as follows.It is assumed that a direction of the viewpoint movement corresponds toa current line of sight. The line of sight depends on head rotation.According to an embodiment, a speed and a direction of the movement mayfollow a touch input on the X axis 1211 or the Y axis 1212 of the touchpad. For example, putting a finger on the X axis 1122 may be analyzed asan intention to move the user along an axis which the user faces theobject. Moving back means inversely moving the viewpoint. The Y axis1212 may be used to move the user on an axis perpendicular to the X axis1211. For example, the X axis 1211 may be used for movement in aforward-backward direction, and the Y axis 1212 may be used of a heightchange. According to another embodiment, a speed and a direction of themovement may follow the up and down rotation 1221 or the left and rightrotation 1222. An embodiment for viewpoint movement within the VR spaceis described below with reference to FIG. 13A.

FIG. 13A is a flowchart illustrating a process in which an electronicdevice moves a viewpoint within a VR space according to an embodiment ofthe disclosure. FIG. 13A illustrates a method of operating theelectronic device 100.

FIG. 13B illustrates a viewpoint movement within a VR space in anelectronic device according to an embodiment of the disclosure, FIG. 13Cillustrates a viewpoint movement within a VR space in an electronicdevice according to an embodiment of the disclosure, and FIG. 13Dillustrates a viewpoint movement within a VR space in an electronicdevice according to an embodiment of the disclosure.

Referring to FIG. 13A, in operation 1301, the electronic devicegenerates and displays a VR space. The electronic device may generate aVR space including at least one object, determine a viewpoint within thespace, and then display a screen based on the determined viewpoint.

In operation 1303, the electronic device identifies an object(hereinafter, referred to as a “reference object”) which is a referenceof viewpoint movement using an eye gaze. The electronic device may trackan eye gaze of the user and identify an object within a VR space whichthe eye gaze faces. According to an embodiment, when the eye gaze of theuser is maintained at a specific object for a predetermined time orlonger, the electronic device may determine the specific object as thereference object. According to another embodiment, when an additionalcommand for designating the reference object is generated while the eyegaze of the user is maintained at the specific object, the electronicdevice may determine the specific object as the reference object. Forexample, the additional command may include at least one of a touchinput (for example, touch input of two points) in a predefined pattern,head rotation (for example, nodding) in a predefined pattern, and avoice command (for example, “Move the viewpoint!”) having the predefinedcontent. A first axis connecting the user and the reference object maybe configured according to designation of the reference object. Thefirst axis is configured as a straight line connecting the user and thereference object and is used as a path along which the viewpoint ismoved by the following input.

In operation 1305, the electronic device identifies whether a commandfor movement on the first axis is detected. According to variousembodiments, the command for movement on the first axis may include atleast one of a first command designating a means indicating a movementdirection and a second command indicating a direction. For example, avoice command 1312 (for example, “Move through head rotation!”) of theuser as illustrated in FIG. 13B may be treated as the first commanddesignating head rotation as the means for inputting the movementdirection. Alternatively, another voice command (for example, “Movethrough the touch!”) of the user may be treated as the first commanddesignating a touch input using a touch panel as the means for inputtingthe movement direction. The movement of the touch point on the X axis1211 of FIG. 12A or the up and down rotation 1221 of FIG. 12B may betreated as the second command indicating the movement directionaccording to the designated means. According to an embodiment, when themeans indicating the direction is predefined, the first command may beomitted.

When the command for movement on the first axis is detected, theelectronic device moves the viewpoint on the first axis within the VRspace in operation 1307. For example, referring to FIG. 13C, theelectronic device may move the viewpoint on a first axis 1350 a. At thistime, the electronic device may move the viewpoint in a direction closerto the object 1310 or farther from the object 1310 according to thedirection indicated by the command. A speed of the movement may dependon a speed of the rotation or a speed of the movement of the touchpoint.

In operation 1309, the electronic device identifies whether a commandfor movement on the second axis is detected. The second axis is acircular axis based on the reference object and contacts the location ofthe user. According to various embodiments, the command for movement onthe second axis may include at least one of a first command designatinga means indicating a movement direction and a second command indicatinga direction. For example, the means for inputting the movement directionmay be designated as head rotation by the voice command 1312 (forexample, “Move through head rotation!”) of the user as illustrated inFIG. 13B. In another example, the means for inputting the movementdirection may be designated as a touch input using a touch panel byanother voice command (for example, “Move through the touch!”) of theuser. Further, the second command indicating the movement direction maybe detected by the movement of the touch point on the Y axis 1212 ofFIG. 12A or the left and right rotation 1222 of FIG. 12B according tothe designated means. According to an embodiment, when the meansindicating the direction is predefined, the first command may beomitted.

When the command for movement on the second axis is detected, theelectronic device moves the viewpoint on the second axis within the VRspace in operation 1311. For example, referring to FIG. 13C, theelectronic device may move the viewpoint on a second axis 1350 b. Atthis time, the electronic device may move the viewpoint in a clockwisedirection or a counterclockwise direction according to the directionindicated by the command. A speed of the movement may depend on a speedof the rotation or a speed of the movement of the touch point.

In the embodiment described referring to FIG. 13A, the movement of theviewpoint is performed by the touch input on the touch panel or the headrotation. According to another embodiment, the touch input on the touchpanel and the head rotation may be combined and used. For example, themovement direction may be determined by one of the touch input and thehead rotation, and the movement speed may be determined by the remainingone. Specifically, the movement direction may be selected by the touchinput (for example, movement of the touch point on the X axis or the Yaxis) on the touch panel, and the movement speed may be selected by thehead rotation (for example, up and down rotation).

In the embodiment described referring to FIG. 13A, each of the commandfor movement on the first axis and the command for movement on thesecond axis may include the first command designating the meansindicating the movement direction. However, when the user inputs aplurality of commands for movement on the first axis or the second axis,only a minimum of the command may include the first command. In thiscase, in at least one following command, the second command indicatingthe direction may follow the means designated by the first commandincluded in an initial command.

In the embodiment described referring to FIG. 13A, the first commanddesignating the means indicating the movement direction has beendescribed as a part of the command for moving the viewpoint. Accordingto another embodiment, the first command may be used as a commanddesignating the reference object. For example, when the first command(for example, the voice command 1312 of FIG. 13B) is generated while theeye gaze of the user is maintained at the specific object, theelectronic device may determine the specific object as the referenceobject and simultaneously determine the means indicating the movementdirection.

As a second lower mode for the control in the VR space, the object editin the VR space may be processed as follows. The object edit may be oneof object addition, object rotation, and object movement. The locationto which the object is added and the object to be rotated/moved may beidentified using the eye gaze, and the object to be added may beselected or the object may be rotated or moved using the following touchinput or head rotation. An embodiment for the object edit within the VRspace is described below referring to FIG. 14A.

FIG. 14A is a flowchart illustrating a process in which an electronicdevice controls an object within a VR space according to an embodimentof the disclosure. FIG. 14A illustrates a method of operating theelectronic device 100.

FIG. 14B illustrates an object control within a VR space in anelectronic device according to an embodiment of the disclosure, FIG. 14Cillustrates an object control within a VR space in an electronic deviceaccording to an embodiment of the disclosure, and FIG. 14D illustratesan object control within a VR space in an electronic device according toan embodiment of the disclosure.

Referring to FIG. 14A, in operation 1401, the electronic devicegenerates and displays a VR space. The electronic device may generate aVR space including at least one object, determine a viewpoint within thespace, and then display a screen based on the determined viewpoint.

In operation 1403, the electronic device identifies an object or alocation within the space based on an eye gaze. According to anembodiment, when an eye gaze of the user is maintained at a specificlocation or a specific object for a predetermined time or longer, theelectronic device may determine the specific location or the specificobject as a location (hereinafter, referred to as a “target location) towhich the specific object is added or an object (hereinafter, referredto as a “target object”) to be moved/rotated. According to anotherembodiment, when an additional command for designating the targetlocation or the target object is generated while the eye gaze of theuser is maintained at the specific location or the specific object, theelectronic device may determine the specific location or the specificobject as the target location or the target object. For example, theadditional command may include at least one of a touch input (forexample, a touch input of two points) in a predefined pattern, headrotation (for example, nodding) in a predefined pattern, and a voicecommand (for example, “Edit the object!”) having the predefined content.

In operation 1405, the electronic device identifies whether a commandfor inserting the object is detected. According to an embodiment, thecommand for inserting the object may include selection of an object tobe inserted. For example, in response to a voice command “Select theobject!” 1412 a as illustrated in FIG. 14B, the electronic device maydisplay a list indicating at least one object which can be added andidentify user's selection for one object included in the list (forexample, using at least one of an eye gaze, a voice command, a touchinput, and head rotation). According to another embodiment, when anobject to be added is predefined or pre-selected, the voice command“Select the object!” 1412 a may be treated as the command for insertingthe object.

When the command for inserting the object is detected, the electronicdevice inserts the selected object to the identified location inoperation 1407. In other words, the electronic device may insert theobject indicated by the user's selection detected in operation 1405 tothe target location identified in operation 1403.

In operation 1409, the electronic device identifies whether a commandfor moving or rotating the object is detected. The command for movementor rotation includes at least one of a first command indicating movementor rotation and a second command indicating a direction of the movementor rotation. The first command may include a voice command. For example,a voice command 1412 b (for example, “Translate!”) of the user formoving the object as illustrated in FIG. 14C may be treated as the firstcommand. In another example, a voice command 1412 c (for example,“Rotate!”) of the user for rotating the object as illustrated in FIG.14D may be treated as the first command indicating rotation. Headrotation followed by the first command may be treated as the secondcommand. According to another embodiment, instead of or in parallel withthe head rotation, a touch input using a touch pad may be treated as thesecond command.

In operation 1411, the electronic device moves or rotates the object ina head rotation direction. For example, when movement is designated asillustrated in FIG. 14C, the electronic device may move an object 1410on a basis of left and right rotation 1421 a along a left and right axis1422 a and move the object 1410 based on an up and down rotation 1421 balong an up and down axis 1422 b. In another example, when rotation isdesignated as illustrated in FIG. 14D, the electronic device may movethe object based on the left and right rotation 1421 a along ahorizontal axis 1423 a and move the object 1410 based on the up and downrotation 1421 b along a vertical axis 1423 b. According to anotherembodiment, the electronic device may move or rotate the objectaccording to a touch input of which a touch point moves along the X axis1211 of FIG. 12A or the Y axis 1212 of FIG. 12A. At this time, amovement speed of the object 1410 may depend on a speed of head rotationor a speed of movement of the touch point.

The above-described various embodiments may be expanded to control theAR object in the AR system. For example, the user may rotate or move theAR object according to the following various embodiments. Embodimentsfor rotating the AR object are described below.

FIG. 15A is flowchart illustrating a process in which an electronicdevice rotates an AR object according to an embodiment of thedisclosure. FIG. 15A illustrates a method of operating the electronicdevice 100.

FIG. 15B illustrates a rotation of an AR object in an electronic deviceaccording to an embodiment of the disclosure.

Referring to FIG. 15A, in operation 1501, the electronic devicegenerates and displays an AR object. In other words, the electronicdevice may generate a virtual AR object and display the generated ARobject on a display unit. For example, the electronic device may detecta real space or a real world through a vision sensor, such as a camera,an Infrared (IR) sensor, a depth sensor, or a depth camera and add avirtual AR object to the detected real space.

In operation 1503, the electronic device identifies whether a commandfor switching the AR object to a movable and rotatable state(hereinafter, a “state switching command”) is detected. The stateswitching command may be variously defined. According to variousembodiments, the state switching command may be defined based on atleast one of an eye gaze, a touch input, a voice command, and a gesture.For example, referring to FIG. 15B, a gesture of making a user's bodypart (for example, hand) approach an AR object 1510 may be treated as astate switching command for the AR object 1510. Specifically, a spaceoccupied by the shape of the AR object 1510 and a point which the bodypart contacts may be understood as at least one anchor point, and thegeneration of at least one anchor point on the surface of or inside theAR object 1510 may be treated as the generation of the state switchingcommand for the AR object 1510. In other words, the electronic devicemay determine at least one anchor point based on a gesture of bring theuser's body part into contact with the space occupied by the AR object.

When the command for switching the AR object to the movable androtatable state is detected, the electronic device identifies whether acommand for moving and rotating the AR object is detected in operation1505. For example, the command for movement and rotation may be definedbased on at least one of an eye gaze, a touch input, a voice command,and a gesture. For example, when the AR object is switched to themovable and rotatable state based on at least one anchor point generatedby the gesture as illustrated in FIG. 15B, movement of at least oneanchor point may be treated as the command for moving and rotating theAR object.

In operation 1507, the electronic device moves or rotates the AR objectaccording to the detected command. The electronic device moves orrotates the AR object according to the detected command for movement androtation in operation 1503. For example, as at least one anchor pointmoves or rotates, the electronic device may move or rotates the ARobject. At this time, the location of the AR object on the screen of theelectronic device may depend on not only the command for movement androtation but also a range of the real space detected by the visionsensor, for example, an angle of a camera. For example, when the angleof the camera moves from left to right even though a command only forrotation is generated, the location of the AR object on the screen maymove from left to right.

The AR object is displayed on a given coordinate of the real space tomake the user experience like being in the real space. For example, whenthe AR object is composed within an image detected by the vision sensoror a space projected and shown on a display unit (for example, lens) ofthe electronic device, the location of the AR object depends on thecoordinate of the real space. For example, when the range of the realspace detected by the vision sensor or captured by the camera ischanged, in other words, when the angle of the camera is changed, thelocation of the AR object on the screen is also changed. If the cameraangle is significantly changed, the AR object may disappear out of thescreen. However, the user may hope to continuously see the correspondingAR object within the screen. Accordingly, the disclosure presents anembodiment of fixing the AR object to a specific location on the screen.

FIG. 16A is a flowchart illustrating a process in which an electronicdevice rotates an AR object according to an embodiment of thedisclosure. FIG. 16A illustrates a method of operating the electronicdevice 100.

FIG. 16B illustrates a rotation of an AR object in an electronic deviceaccording to an embodiment of the disclosure.

Referring to FIG. 16A, in operation 1601, the electronic devicegenerates and displays an AR object. In other words, the electronicdevice may generate and display a virtual AR object. For example, theelectronic device may detect a real space through the vision sensor andadd the virtual AR object to the detected real space.

In operation 1603, the electronic device identifies whether a commandfor fixing the location of the AR object on the screen (hereinafter,referred to as a “fixing command”) is detected. The fixing command maybe variously defined. According to various embodiments, the fixingcommand may be defined based on at least one of an eye gaze, a touchinput, a voice command, and a gesture. For example, a gesture of makinga user's body part (for example, hand) approach the AR object for apredetermined time or longer may be treated as the fixing command.Specifically, a space occupied by the shape of the AR object and a pointwhich the body part contacts for the predetermined time or longer may beunderstood as an anchor point, and the generation of the anchor point onthe surface of or inside the AR object may be treated as the generationof the fixing command for the corresponding AR object.

When the fixing command for the AR object is detected, the electronicdevice fixes the location of the AR object on the screen in operation1605. The fixed location may be a predefined location or a location ofthe AR object displayed on the real space detected when the fixingcommand is generated. At this time, the electronic device may detect adepth of the AR object in the real space, a distance from the electronicdevice, and a distance or direction relative to the object in the realworld. When the eye gaze of the user or the electronic device movesafter the AR object is fixed in the real space according to the fixingcommand, the electronic device may process as if the AR object is fixedin the real space based on the eye gaze or the size, direction, andangle change of the movement of the electronic device. For example, whenthe AR object has a three-dimensional form and is fixed to the realspace, another side of the three-dimensional AR object may be displayedaccording to movement of the user or the electronic device. Accordingly,even though the camera angle is changed, the AR object is displayed atthe fixed location in the real space projected on or by the display unitof the electronic device before the fixing command is released. Forexample, referring to FIG. 16B, an AR object 1610 may be displayed onthe upper right part of the screen.

In operation 1607, the electronic device rotates the AR object at thefixed location according to a change in a capturing range. The locationof the AR object on the screen may be fixed by the fixing command, orrotation of the AR object is allowed. Accordingly, in response to thechange in the camera angle, the AR object may be rotated. For example,referring to FIG. 16B, when the camera angle is rotated to the left fromthe front, the AR object 1610 is still displayed on the upper right partof the screen but the AR object 1610 is rotated to the left. Similarly,referring to FIG. 16B, when the camera angle is rotated to the rightfrom the front, the AR object 1610 is still displayed on the upper rightpart of the screen but the AR object 1610 is rotated to the right.

Various embodiments for controlling the AR object described withreference to FIGS. 15A to 16B may be applied to various applications.For example, various embodiments for controlling the AR object may beapplied to an application for displaying the virtual AR objectindicating a finished product or an assembly manual while prefabricatedmodels are assembled. In this case, the user may also see the shape ofthe finished product while assembling the prefabricated models, and maysee the finished product in various angles by moving, fixing, orrotating the AR object (for example, using the state switching commandor the fixing command) according to various embodiments as necessary.

The above-described various embodiments may be implemented based onvarious types of inputs. In order to process various types of inputs,the electronic device is required to have a corresponding capability.

For example, an eye gaze tracking function may be needed. The eye gazetracking technology may be performed by various devices. The eye gazetechnology may be implemented using an infrared (IR) signal and apattern or based on image analysis. The electronic device may performeye gaze tracking by analyzing movement of a user's eyes.

In another example, a technology for estimating a distance between anobject existing in a real space or a virtual space and the electronicdevice may be required. For example, the distance may be estimated usingimages captured through two cameras. The electronic device may obtaindisparity for the object and estimate the distance based on thedisparity. Alternatively, a depth camera technology may be used toestimate the same.

In another example, a technology for detecting a gesture or headrotation may be required. Physical movement of the gesture and the headrotation may be detected through a sensor. The electronic device maydetect the existence and size of physical movement of the gesture andthe head rotation by analyzing measurement values provided from sensors(for example, an acceleration sensor, a gyro sensor, and a gravitysensor).

Methods according to embodiments stated in claims and/or specificationsof the disclosure may be implemented in hardware, software, or acombination of hardware and software.

When the methods are implemented by software, a computer-readablestorage medium for storing one or more programs (software modules) maybe provided. The one or more programs stored in the computer-readablestorage medium may be configured for execution by one or more processorswithin the electronic device. The at least one program may includeinstructions that cause the electronic device to perform the methodsaccording to various embodiments of the disclosure as defined by theappended claims and/or disclosed herein.

The programs (software modules or software) may be stored innon-volatile memories including a random access memory and a flashmemory, a Read Only Memory (ROM), an Electrically Erasable ProgrammableRead Only Memory (EEPROM), a magnetic disc storage device, a CompactDisc-ROM (CD-ROM), Digital Versatile Discs (DVDs), or other type opticalstorage devices, or a magnetic cassette. Alternatively, any combinationof some or all of the memory described above may form a memory in whichthe program is stored. Further, a plurality of such memories may beincluded in the electronic device.

In addition, the programs may be stored in an attachable storage devicewhich is accessible through communication networks, such as theInternet, Intranet, local area network (LAN), wide area network (WAN),and storage area network (SAN), or a combination thereof. Such a storagedevice may access the electronic device via an external port. Further, aseparate storage device on the communication network may access aportable electronic device.

In the above-described detailed embodiments of the disclosure, acomponent included in the disclosure is expressed in the singular or theplural according to a presented detailed embodiment. However, thesingular form or plural form is selected for convenience of descriptionsuitable for the presented situation, and various embodiments of thedisclosure are not limited to a single element or multiple elementsthereof. Further, either multiple elements expressed in the descriptionmay be configured into a single element or a single element in thedescription may be configured into multiple elements.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. A method performed by an electronic device, themethod comprising: detecting maintenance of an eye gaze of a usercorresponding to an object in a real space; generate a virtual AugmentedReality (AR) object corresponding to the object in the real space inresponse to the eye gaze being maintained for a predetermined time orlonger; displaying the AR object corresponding to a first angle;determining at least one anchor point by designating a target to becontrolled, based on a first input including at least one of a firstgesture, a voice input, or an eye gaze, wherein the at least one anchorpoint designates the AR object; identifying an operation to be performedfor the target designated by the at least one anchor point, based on asecond input including a head rotation, wherein the second inputincludes a second gesture of changing an angle of a vision sensorincluded in the electronic device; performing the operation whichrotates the AR object in a direction corresponding to the head rotationat a fixed location while fixing a location of the AR object in the realspace regardless of the change in the angle of the vision sensor,wherein the AR object is rotated based on the at least one anchor point;and displaying the AR object corresponding to a second angle differentfrom the first angle, in response to the rotation of the AR object. 2.The method of claim 1, wherein each of the first input and the secondinput further includes at least one of an eye gaze tracking result, agesture, a voice, or a touch input, and wherein the first input and thesecond input include different types of inputs from each other.
 3. Themethod of claim 1, wherein the at least one anchor point is determinedby the first gesture of bringing a user's body part into contact with aspace occupied by a virtual Augmented Reality object, and wherein theoperation moves or rotates the AR object as the at least one anchorpoint is moved or rotated by the second gesture.
 4. The method of claim1, wherein the first input includes maintenance of an eye gaze at aspecific point of an image for a predetermined time or longer, whereinthe second input includes a touch input, wherein the at least one anchorpoint designates the specific point, and wherein the operation modifiesthe image, based on the specific point.
 5. An electronic devicecomprising: a detector configured to detect a state inside or outsidethe electronic device; and at least one processor connected to thedetector, wherein the at least one processor is configured to: detectmaintenance of an eye gaze of a user corresponding to an object in areal space, generate a virtual Augmented Reality (AR) objectcorresponding to the object in the real space in response to the eyegaze being maintained for a predetermined time or longer, display the ARobject corresponding to a first angle, determine at least one anchorpoint by designating a target to be controlled, based on a first inputidentified by the detector, wherein the first input includes at leastone of a first gesture, a voice input, or an eye gaze and the at leastone anchor point designates the AR object, identify an operation to beperformed for the target designated by the at least one anchor point,based on a second input identified by the detector, wherein the secondinput includes a head rotation and a second gesture of changing an angleof a vision sensor included in the electronic device, perform theoperation which rotates the AR object in a direction corresponding tothe head rotation at a fixed location while fixing a location of the ARobject in the real space regardless of the change in the angle of thevision sensor, wherein the AR object is rotated based on the at leastone anchor point; and display the AR object corresponding to a secondangle different from the first angle, in response to the rotation of theAR object.
 6. The electronic device of claim 5, wherein each of thefirst input and the second input further includes at least one of an eyegaze tracking result, a gesture, a voice, or a touch input, and whereinthe first input and the second input include different types of inputsfrom each other.
 7. The electronic device of claim 5, wherein the atleast one anchor point is determined by the first gesture of bringing auser's body part into contact with a space occupied by a virtualAugmented Reality object, and wherein the operation moves or rotates theAR object as the at least one anchor point is moved or rotated by thesecond gesture.
 8. The electronic device of claim 5, wherein the firstinput includes maintenance of an eye gaze at a specific point of animage for a predetermined time or longer, wherein the second inputincludes a touch input, wherein the at least one anchor point designatesthe specific point, and wherein the operation modifies the image, basedon the specific point.
 9. The electronic device of claim 5, wherein thefirst input includes maintenance of an eye gaze at a specific point ofan image for a predetermined time or longer, wherein the at least oneanchor point designates the specific point, and wherein the operationrotates the image, based on the specific point according to the headrotation.
 10. The electronic device of claim 5, wherein the first inputincludes an eye gaze at a specific point of a preview image of an imageacquired through a camera, wherein the second input includes maintenanceof the eye gaze at the specific point for a predetermined time orlonger, wherein the at least one anchor point designates the specificpoint, and wherein the operation controls a focal distance of thecamera, based on the specific point.
 11. The electronic device of claim5, wherein the first input includes maintenance of an eye gaze of a userat a part of a virtual Augmented Reality object corresponding to anobject in a real space generated by the eye gaze of the user at theobject in the real space for a predetermined time or longer, wherein theat least one anchor point designates the part, and wherein the operationcontrols attributes of the part to be a value indicated by the headrotation.
 12. The electronic device of claim 5, wherein the first inputincludes at least one of an eye gaze of a user at a virtual objectwithin a Virtual Reality (VR) space or a voice command, wherein the atleast one anchor point designates the virtual object, and wherein theoperation moves a viewpoint within the VR space, based on the virtualobject.
 13. The electronic device of claim 5, wherein the first inputincludes at least one of an eye gaze of a user at a location within aVirtual Reality (VR) space or a voice command, wherein the at least oneanchor point designates the location, and wherein the operation includesinsertion of an object into the location.
 14. The electronic device ofclaim 5, wherein the first input includes at least one of an eye gaze ofa user at an object within a Virtual Reality (VR) space or a voicecommand, wherein the at least one anchor point designates the object,and wherein the operation moves the object within the VR space.
 15. Theelectronic device of claim 5, wherein the at least one anchor pointincludes one of an object displayed on the electronic device or onepoint on the object.
 16. The electronic device of claim 5, wherein anobject includes one of a content, attributes of the content, anapplication, a parameter related to the application, or an element ofthe electronic device.
 17. The electronic device of claim 5, wherein theprocessor is further configured to: detect a characteristic of an objectin a real space based on an image of the object captured by a camera,the characteristic comprising an object type.
 18. The electronic deviceof claim 17, wherein the object type comprises one of a person, a dog, acat, or a flower.